Skip to content

Conversation

@GrigoriyPA
Copy link
Collaborator

Changelog entry

Supported stream lookup join with external data sources in YDB

Changelog category

  • New feature

Description for reviewers

@GrigoriyPA GrigoriyPA requested review from a team as code owners September 26, 2025 15:24
@GrigoriyPA GrigoriyPA marked this pull request as draft September 26, 2025 15:24
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements stream lookup join functionality with external data sources in YDB. The feature enables efficient lookup joins where streaming data from one source is joined with lookup data from external sources like S3 or generic connectors.

Key changes include:

  • Refactored stream lookup join logic to common DQ optimization modules
  • Added support for external data sources in streaming queries
  • Implemented caching and validation mechanisms for stream lookup operations

Reviewed Changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
ydb/library/yql/dq/opt/dq_opt_phy.cpp Moved stream lookup join rewrite logic to common DQ optimization module
ydb/library/yql/dq/opt/dq_opt_join.cpp Moved stream equi-join with lookup rewrite logic to common DQ join module
ydb/core/protos/kqp_physical.proto Added new TKqpPhyCnDqSourceStreamLookup protobuf message for stream lookup connections
ydb/core/kqp/query_compiler/kqp_query_compiler.cpp Added stream lookup connection handling in query compilation
ydb/library/yql/providers/generic/connector/libcpp/ut_helpers/connector_client_mock.h Enhanced mock connector client to support multiple responses and validation options
ydb/core/kqp/ut/federated_query/datastreams/datastreams_ut.cpp Added comprehensive tests for stream lookup join functionality
Comments suppressed due to low confidence (2)

ydb/core/kqp/ut/federated_query/datastreams/datastreams_ut.cpp:1

  • Remove or update the outdated comment '// temp replace with yt source' which doesn't reflect the actual functionality of this function that handles generic lookup sources.
#include <ydb/core/external_sources/external_source.h>

ydb/core/kqp/ut/federated_query/datastreams/datastreams_ut.cpp:1

  • The removal of FillWithDefaults() call from constructor creates inconsistency. Either remove the FillWithDefaults() method entirely or ensure it's called consistently across similar classes in this file.
#include <ydb/core/external_sources/external_source.h>

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@github-actions
Copy link

github-actions bot commented Sep 26, 2025

2025-09-26 15:26:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for cb0f56f has started.
2025-09-26 15:26:31 UTC Artifacts will be uploaded here
2025-09-26 15:30:31 UTC ya make is running...
🔴 2025-09-26 16:22:35 UTC Build failed, see the logs. Also see fail summary
🟡 2025-09-26 16:22:47 UTC ydbd size 2.2 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 3db6964 merge: cb0f56f diff diff %
ydbd size 2 395 787 520 Bytes 2 396 935 000 Bytes +1.1 MiB +0.048%
ydbd stripped size 512 340 680 Bytes 512 499 464 Bytes +155.1 KiB +0.031%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 26, 2025

2025-09-26 15:27:04 UTC Pre-commit check linux-x86_64-release-asan for cb0f56f has started.
2025-09-26 15:27:08 UTC Artifacts will be uploaded here
2025-09-26 15:31:15 UTC ya make is running...
🔴 2025-09-26 16:28:10 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

🟢 2025-09-26 15:27:54 UTC The validation of the Pull Request description is successful.

@GrigoriyPA GrigoriyPA requested a review from Copilot September 26, 2025 16:46
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@github-actions
Copy link

github-actions bot commented Sep 26, 2025

2025-09-26 16:49:25 UTC Pre-commit check linux-x86_64-release-asan for e686d60 has started.
2025-09-26 16:50:13 UTC Artifacts will be uploaded here
2025-09-26 16:54:53 UTC ya make is running...
2025-09-26 18:00:33 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 26, 2025

2025-09-26 16:49:32 UTC Pre-commit check linux-x86_64-relwithdebinfo for e686d60 has started.
2025-09-26 16:50:14 UTC Artifacts will be uploaded here
2025-09-26 16:54:22 UTC ya make is running...
2025-09-26 18:00:33 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 26, 2025

2025-09-26 18:02:12 UTC Pre-commit check linux-x86_64-relwithdebinfo for bc4b96d has started.
2025-09-26 18:02:55 UTC Artifacts will be uploaded here
2025-09-26 18:07:35 UTC ya make is running...
🟡 2025-09-26 19:28:13 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40747 37980 0 6 2728 33

2025-09-26 19:28:23 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-09-26 19:41:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
250 (only retried tests) 227 0 2 0 21

2025-09-26 19:41:29 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-09-26 19:52:18 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
218 (only retried tests) 198 0 2 0 18

🟢 2025-09-26 19:52:22 UTC Build successful.
🟢 2025-09-26 19:52:39 UTC ydbd size 2.2 GiB changed* by -41.0 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 580503a merge: bc4b96d diff diff %
ydbd size 2 397 008 592 Bytes 2 396 966 568 Bytes -41.0 KiB -0.002%
ydbd stripped size 512 481 192 Bytes 512 504 840 Bytes +23.1 KiB +0.005%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA requested a review from uzhastik September 26, 2025 18:04
@github-actions
Copy link

github-actions bot commented Sep 26, 2025

2025-09-26 18:05:03 UTC Pre-commit check linux-x86_64-release-asan for bc4b96d has started.
2025-09-26 18:05:35 UTC Artifacts will be uploaded here
2025-09-26 18:10:28 UTC ya make is running...
🟡 2025-09-26 20:15:28 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17783 17314 0 192 244 33

🟢 2025-09-26 20:15:33 UTC Build successful.
🟢 2025-09-26 20:15:57 UTC ydbd size 3.7 GiB changed* by -111.0 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 580503a merge: bc4b96d diff diff %
ydbd size 3 986 662 296 Bytes 3 986 548 664 Bytes -111.0 KiB -0.003%
ydbd stripped size 1 485 964 128 Bytes 1 486 036 992 Bytes +71.2 KiB +0.005%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA added the rebase-and-check Rebase PR with the current base branch and check label Sep 29, 2025
@github-actions github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Sep 29, 2025
@github-actions
Copy link

github-actions bot commented Sep 29, 2025

2025-09-29 09:06:24 UTC Pre-commit check linux-x86_64-release-asan for 4860041 has started.
2025-09-29 09:06:41 UTC Artifacts will be uploaded here
2025-09-29 09:10:43 UTC ya make is running...
🟡 2025-09-29 11:50:49 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17777 17383 0 111 256 27

🟢 2025-09-29 11:51:00 UTC Build successful.
🟢 2025-09-29 11:51:22 UTC ydbd size 3.7 GiB changed* by -115.0 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 234f6d5 merge: 4860041 diff diff %
ydbd size 3 987 121 616 Bytes 3 987 003 896 Bytes -115.0 KiB -0.003%
ydbd stripped size 1 486 129 568 Bytes 1 486 198 336 Bytes +67.2 KiB +0.005%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 29, 2025

2025-09-29 09:08:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4860041 has started.
2025-09-29 09:09:02 UTC Artifacts will be uploaded here
2025-09-29 09:13:58 UTC ya make is running...
🟡 2025-09-29 11:24:06 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40749 37866 0 6 2844 33

2025-09-29 11:24:16 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-09-29 11:40:41 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
498 (only retried tests) 475 0 2 0 21

2025-09-29 11:40:44 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-09-29 11:51:41 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
215 (only retried tests) 196 0 2 0 17

🟢 2025-09-29 11:51:43 UTC Build successful.
🟢 2025-09-29 11:52:02 UTC ydbd size 2.2 GiB changed* by -44.9 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 234f6d5 merge: 4860041 diff diff %
ydbd size 2 397 243 112 Bytes 2 397 197 088 Bytes -44.9 KiB -0.002%
ydbd stripped size 512 537 768 Bytes 512 557 416 Bytes +19.2 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@yumkam
Copy link
Collaborator

yumkam commented Sep 29, 2025

Тесты на slj в ydb/tests/fq/generic/streaming надо запускать руками (large)

ya test --buid relwithdebinfo -ttt ydb/tests/fq/generic/streaming

@github-actions
Copy link

github-actions bot commented Sep 30, 2025

2025-09-30 09:58:18 UTC Pre-commit check linux-x86_64-release-asan for 647ef53 has started.
2025-09-30 09:58:33 UTC Artifacts will be uploaded here
2025-09-30 10:02:42 UTC ya make is running...
2025-09-30 10:46:27 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 30, 2025

2025-09-30 09:58:48 UTC Pre-commit check linux-x86_64-relwithdebinfo for 647ef53 has started.
2025-09-30 09:59:03 UTC Artifacts will be uploaded here
2025-09-30 10:03:05 UTC ya make is running...
2025-09-30 10:46:27 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Oct 6, 2025

2025-10-06 15:37:09 UTC Pre-commit check linux-x86_64-relwithdebinfo for 09f7aa6 has started.
2025-10-06 15:37:22 UTC Artifacts will be uploaded here
2025-10-06 15:41:32 UTC ya make is running...
🟡 2025-10-06 17:37:01 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40895 38107 0 3 2756 29

2025-10-06 17:37:12 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-10-06 17:51:30 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
532 (only retried tests) 494 0 2 18 18

2025-10-06 17:51:33 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-10-06 18:03:57 UTC Tests successful.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
573 (only retried tests) 540 0 0 16 17

🟢 2025-10-06 18:04:00 UTC Build successful.
🟢 2025-10-06 18:04:18 UTC ydbd size 2.3 GiB changed* by -40.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: a20ebf0 merge: 09f7aa6 diff diff %
ydbd size 2 416 436 472 Bytes 2 416 395 008 Bytes -40.5 KiB -0.002%
ydbd stripped size 514 853 928 Bytes 514 877 224 Bytes +22.8 KiB +0.005%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Oct 6, 2025

2025-10-06 15:37:15 UTC Pre-commit check linux-x86_64-release-asan for 09f7aa6 has started.
2025-10-06 15:37:36 UTC Artifacts will be uploaded here
2025-10-06 15:42:17 UTC ya make is running...
🟡 2025-10-06 18:02:01 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17893 17528 0 134 214 17

🟢 2025-10-06 18:02:08 UTC Build successful.
🟢 2025-10-06 18:02:30 UTC ydbd size 3.7 GiB changed* by -661.1 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: a20ebf0 merge: 09f7aa6 diff diff %
ydbd size 4 019 476 904 Bytes 4 018 799 888 Bytes -661.1 KiB -0.017%
ydbd stripped size 1 493 478 784 Bytes 1 493 542 816 Bytes +62.5 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

APozdniakov
APozdniakov previously approved these changes Oct 7, 2025
vitstn
vitstn previously approved these changes Oct 7, 2025
@GrigoriyPA GrigoriyPA dismissed stale reviews from vitstn and APozdniakov via 5add164 October 7, 2025 09:51
@GrigoriyPA GrigoriyPA force-pushed the YQ-4602-support-stream-lookup-join branch from 13df35b to 5add164 Compare October 7, 2025 09:51
@GrigoriyPA GrigoriyPA requested review from APozdniakov and vitstn and removed request for a team October 7, 2025 09:52
@github-actions
Copy link

github-actions bot commented Oct 7, 2025

2025-10-07 09:53:39 UTC Pre-commit check linux-x86_64-relwithdebinfo for 53eb297 has started.
2025-10-07 09:53:43 UTC Artifacts will be uploaded here
2025-10-07 09:57:50 UTC ya make is running...
🟡 2025-10-07 12:04:08 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40899 38098 0 5 2767 29

2025-10-07 12:04:18 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-10-07 12:30:02 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
809 (only retried tests) 788 0 0 1 20

🟢 2025-10-07 12:30:07 UTC Build successful.
🟢 2025-10-07 12:30:23 UTC ydbd size 2.3 GiB changed* by -35.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 7d731ec merge: 53eb297 diff diff %
ydbd size 2 416 668 600 Bytes 2 416 632 280 Bytes -35.5 KiB -0.002%
ydbd stripped size 514 878 056 Bytes 514 902 504 Bytes +23.9 KiB +0.005%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Oct 7, 2025

2025-10-07 09:55:48 UTC Pre-commit check linux-x86_64-release-asan for 53eb297 has started.
2025-10-07 09:56:02 UTC Artifacts will be uploaded here
2025-10-07 10:00:05 UTC ya make is running...
🟡 2025-10-07 12:43:36 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17897 17497 0 167 218 15

🟢 2025-10-07 12:43:42 UTC Build successful.
🟢 2025-10-07 12:44:05 UTC ydbd size 3.7 GiB changed* by -818.4 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 464a8e4 merge: 53eb297 diff diff %
ydbd size 4 020 037 936 Bytes 4 019 199 880 Bytes -818.4 KiB -0.021%
ydbd stripped size 1 493 577 728 Bytes 1 493 637 600 Bytes +58.5 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@GrigoriyPA GrigoriyPA merged commit d2fb053 into ydb-platform:main Oct 7, 2025
14 checks passed
@GrigoriyPA GrigoriyPA deleted the YQ-4602-support-stream-lookup-join branch October 7, 2025 18:07
neyrox pushed a commit to neyrox/ydb that referenced this pull request Oct 10, 2025
GrigoriyPA added a commit to GrigoriyPA/ydb that referenced this pull request Oct 14, 2025
GrigoriyPA added a commit to GrigoriyPA/ydb that referenced this pull request Oct 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants